Skip to main content

View Post [edit]

Poster: TheXmanCometh Date: Aug 24, 2015 2:51pm
Forum: faqs Subject: Web History for a website with no content

In the past I was able to search history for pages from the website http://charliesangels.org.

About a year ago the site went away but you can try to access it (internal server error.) This happens even if you try to reach a page that you know existed:
http://charliesangels.org/episodes/antique-angels.html

When I search here I receive "Page cannot be crawled or displayed due to robots.txt" and I think this is happening only because robots.txt cannot be accessed. Is there any way you can allow searches for sites like this that are alive but all content was removed? You should be able to rely on the cached version of the file to verify that the site owner wasn't preventing site searches.